DOCUMENT RESUME 



ED 414 899 



IR 018 679 



AUTHOR 

TITLE 

INSTITUTION 

REPORT NO 
PUB DATE 
NOTE 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Abels, Eileen G. ; White, Marilyn Domas ; Hahn, Karla 

The Development and Implementation of a User-Based Design 

Process in Web Site Design. 

Maryland Univ. , College Park. Coll, of Library and 
Information Services. 

CLIS-TR- 97-04 
1997-12-00 

52p . ; Research funded by Disclosure, Inc. 

Reports - Descriptive (141) 

MF01/PC03 Plus Postage. 

Access to Information; ^Business Education; Evaluation 
Methods; *Focus Groups; Higher Education; Information 
Retrieval; * Information Seeking; Information Systems; 
Information Utilization; Online Systems; User Needs 
(Information); *Users (Information); *World Wide Web 
*Web Sites 



ABSTRACT 



This report describes the development of a World Wide Web 
site designed for business school faculty. During the project, the project 
team formulated and followed a user-based design process, which places the 
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does not discuss in detail all activities involved in the development. 

Greater attention is given to describing and presenting the results of the 
data-gathering phase and to explaining the user-based design process both in 
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supplementary questionnaires. It emphasized identifying the criteria that 
business faculty consider in using a Web site but also placed these within 
the context of general information-related behavior. Appendices include human 
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This report describes the development of a World Wide Web site designed for 
business school faculty. During the project, the project team formulated and followed a 
user-based design process, which places the user centrally in all stages of the process: 
information-gathering, development, test and evaluation, and implementation. The 
technical report does not discuss in detail all activities involved in the development. 
Greater attention is given to describing and presenting the results of the data-gathering 
phase and to explaining the user-based design process both in general and as it was 
applied in this project. The information-gathering phase is based on a focus group 
session with faculty from four business schools in an electronic environment, using 
VisionQuest software, and on supplementary questionnaires. It emphasized identifying 
the criteria that business faculty consider in using a Web site but also placed these within 
the context of general information-related behavior. 
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Part I. Introduction and Literature Review 

This report describes the development of a World Wide Web site designed for 
business school faculty. During the project, the project team formulated and followed a 
user-based design process. This process is described in greater detail in Part III. The 
technical report does not discuss in detail all activities involved in the development. 
Greater attention is given to describing and presenting the results of the data-gathering 
phase and to presenting the user-based design process both in general and as it was 
applied in this particular project. 

Literature Review 

The emergence of the World Wide Web has resulted in the creation of a wide 
range of new information resources. Perhaps because the Web presents a new medium 
for information provision, these resources vary widely in quality and utility. Both 
resource providers and resource users face the problem of determining the hallmarks of a 
well-designed, useful resource on the Web. Thus users and designers alike continue to 
struggle with the question of how to design pages effectively and how to evaluate 
existing Web-based resources. A number of potential approaches to evaluating Web 
resources have been suggested to date. 

The earliest impulse seems to be developing somewhat ad hoc criteria largely 
based on the features offered by the Web and HTML architectures and the personal 
experiences of designers. The criteria offered vary widely in their usefulness and 
generalizability. A plethora of articles and books offer advice to would-be designers or 
evaluators. For instance, Falcigno and Green (1995) suggest that well-designed sites 
demonstrate a clear understanding of audience needs and preferences, appropriate use of 
hypertext links, reasonable page length, organization, balance between graphics and 
transmission speed, and currency of links. McClements and Becker (1996) suggest that 
links connecting all site pages to the site’s home page, identifying graphics on each page, 
a short home page, links, limited use of graphic, short cuts, and user testing are desirable 
design elements. While this type of advice is helpful to designers, concrete measures of 
successful design are lacking. 

The limitations of using design advice as a basis for the creation of design criteria 
for subsequent evaluation of Web sites is illustrated by Stover and Zink (1996) who 
analyzed 40 Web pages to determine page characteristics such as the number of links, 
number of images, length, image size, date of last modification, presence of statement of 
purpose, links to sponsoring organization, indication of authorship, and presence of a 
comment gathering mechanism on the home page. This list of criteria, which was drawn 
from the literature of advice to designers, was translated into a simple, unweighted point 
system for comparative evaluation of the sites. They interpreted the relatively low scores 
of the sites examined as reflecting general poor design of higher education Web pages. 
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However, the simplicity of their metric and the lack of any theoretical base for then- 
selection of criteria makes interpretation of their results problematic. They offer no 
report of any attempt to validate their metric. 

Another weakness of this type of approach to evaluation is that the developer 
community is consistently the source of this type of the information used. While the 
importance of knowing a site's audience and their needs is frequently emphasized, the 
views of actual users are not incorporated. It is not unlikely that developers’ preferences 
differ in significant ways from those of users and that sites developers consider well- 
designed may be viewed in a very different light by users. 

An alternative approach, championed by some in the library co mmuni ty, is to 
develop evaluation criteria using insights from the evaluation of print-based information 
resources. Pratt, Flannery, and Perkins (1996) advocate this type of approach and provide 
guidelines that include the usual categories of print evaluation criteria, such as content, 
comprehensiveness, uniqueness, authority, and cost, and then suggest additional criteria 
such as need for hardware or software, ease of use, search capabilities, and availability. 
Rettig (1996) also provides a detailed discussion of the relationship between evaluation 
criteria for paper resources and Web resources. 

A third approach draws on research related to computer interface design. The 
interface design community has been quick to realize that much of the knowledge based 
developed for design of other types of information systems, and particularly other types 
of hypertext systems, may be relevant to the issues of Web site design. The subject has 
been discussed at several meetings (Instone, 1996; Shum, 1996). Nielsen and Sano 
(1994) report the results of a project to implement traditional interface design techniques 
into a Web site design. Shneiderman (1996) synthesizes the implications of interface 
design traditions to suggest principles for Web site design. Borges, Morales, and 
Rodriguez (1996) developed a set of guidelines heuristically and report on the results of 
usability testing based on the design criteria they identified. While this approach may 
seem superficially similar to that used by Stover and Zink (1996), the usability testing 
approach employed is widely accepted in the interface design community. Although the 
approach was more rigorous than many others, the testing was performed with a small 
number of subjects and their largely ambiguous findings must be viewed as prel iminar y. 

An alternative approach to the development of design criteria hag been largely 
unexplored. Web site developers and librarians have generated criteria based on then- 
experiences and outlooks. Users are represented at best secondarily in these approaches 
to design criteria development. The importance of users in the development of computer 
systems has been recognized since the 1970s by software engineers, interface designers 
and information professionals; however, efforts made by these groups have not been 
coordinated nor does one group build on lessons learned from the others. For example, 
the results of studies of user needs and information seeking behavior conducted by 
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information professionals (See, for example, Hewins, 1989; Shaw, 1991) have not been 
linked to the work of software engineers and interface designers. 
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Software engineers have recently noted the need to enhance the identification and 
definition of user requirements by working more closely with researchers involved in 
human-computer interaction (Johnson and Jones, 1 997). However, user requirements 
seem to be defined narrowly in terms of tasks. User involvement in the design of 
computer systems has been limited primarily to usability testing conducted during the 
evaluation stage, after a system has been designed. Thus, while it has been recognized 
that an understanding of user behavior is important to designing information systems, 
user behavior has been defined narrowly in terms of specific tasks using task analysis to 
analyze and describe a particular end-user task (Hartson and Hix, 1 989). While tasks are 
important, user requirements based solely on tasks lack the scope necessary to develop a 
truly effective system. 

One exception to this is a move toward participatory design, also referred to as 
cooperative design in parts of Scandinavia. Participatory design was introduced as a 
method of joint design of software products between the user as expert and the system 
designer. This approach goes beyond the user's role in testing an already designed system 
or prototype (Floyd et al., 1989). The importance of considering user needs in the 
development of the Internet, both as a network and in individual Web sites, has been 
noted in the literature. On a broad level, Muller (1996) discusses the need to consider the 
multicultural constituencies using the Internet. Focusing on Web page design, 
Shneiderman (1996) notes that, as in any user interface design process, the initial 
question facing Web site designers relates to the identification of the users and the users’ 
tasks. He indicates that Web pages should be targeted to specific user groups in much the 
same way that advertising is geared to certain groups. One Web site will not meet the 
needs of all potential users of that site, just as one print reference book will not meet the 
needs of all individuals interested in its subject. The idea of segmenting Web pages by 
user groups has not been adopted in many instances but would enhance the effectiveness 
of many Web pages. 

There is little evidence that user needs have been incorporated into the most Web 
site design efforts. Few examples of collaborative or participatory Web page design have 
been found in the literature. One collaborative design described efforts between 
librarians, rather than end-users, and departmental Web designers (Andrew and Musser, 
1997). Neilsen (1995) reports on a case study in which a Web page was designed 
incorporating usability testing. Shneiderman (1996) notes the lack of empirical and 
objective data to assist future designers. The case studies found are anecdotal and do not 
present a methodological and systematic approach 

It may be that the overall lack of user input into design has developed because 
users as a group have relatively limited, idiosyncratic use of Web resources and that most 
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users have not had time to develop the level of use that would allow them to provide 
useful oversight of Web design issues. However, the Web is maturing rapidly and 
currently a body of users has developed possessing the necessary levels of use of Web 
sites and the breadth of exposure to allow the exploration of a user-based approach. 
User-based criteria would fill a gap in the range of approaches to criteria development 
and could provide a rich source of insight into an area that seems to be widely recognized 
as problematic. 

For this project, business school faculty members offer several advantages as a 
participants in the development of a user-centered approach to criteria development. As 
Diamond (1996) reports, business faculty use a wide range of information sources 
including both primary and secondary sources. They often seek information that is 
rapidly outdated and often difficult for libraries to acquire. They often use information 
that is consider valuable by information producers and has a wide audience beyond 
academe. There is evidence that a significant amount of business information is available 
online and has been for a while (Haas, 1994; Morgan and Kelly-Milbum, 1994; Turdor, 
1996). Several projects to disseminate government information relating to securities have 
been highly publicized (Bates, 1996; Pagell, 1995). Thus business faculty could be 
expected to be motivated to seek out and use Web resources and have a significant base 
of expertise as Web users. 

Corporations, government agencies, non-profit groups, and other organizations 
are recognizing the potential of the World Wide Web to disseminate information 
resources. The quality of these sources varies widely, since there is little consensus on 
good design and relatively little standardization. For a Web site to be successful amid the 
millions of sites available, it must satisfy the information needs of its intended audience 
and meet the user’s criteria for repeated use. 

Overall Project Methodology 

In developing the Web site, the Project Team also developed and implemented a 
user-based design process. This process, shown in Figure 1, places the user at the center 
of the design process and emphasizes his participation throughout the initial design and 
development of the site and into its implementation and long-term development. It 
consists of four stages: Information-gathering; Development; Evaluation; and 
Implementation. Part III describes these stages and the sub-activities involved in each 
both generally and specifically in relation to this project. Part II describes in considerable 
detail efforts to gain information about user criteria for evaluating Web site use and the 
relationship between business school faculty’s use of the World Wide Web and their 
other information-gathering behavior. 




Figure 1. User-based Design Process 



4. Implementation Stage 

- Activate continual feedback mechanisms to 
allow users to evaluate the site and suggest 
content or other modifications. 

- Announce the site’s availability to primary user 
groups, using both electronic and non-electronic 
media. 

- Monitor and modify as necessary or desirable. 

Test and Evaluation Stage : 

- Evaluate the Web site and/or features of the site from 
the users’ perspective. 

- Modify the site based on feedback. 

[Repeat process until acceptable version is ready.] 

2. Development Stage 

- Define and/or operationalize the criteria. 

- Assess the implications of the ranked criteria for design. 

- Establish priorities of criteria based on feasibility. 

- Translate the criteria into Web site features. 

- Incorporate feedback mechanisms into design. 

- Design a preliminary version of the Web site.. 

lllMililllllll ill I Information-gathering Stage 

- [Determine the users overall information seeking/use behavior.] 

- Identify user criteria. 

- Rank the criteria. 



Users* Task-related Information Seeking/Use Behavior | 
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Part II. Identification of User-based Design Criteria 
Research Questions 

The research addresses several important questions. The first, and most 
important, focuses on identifying the criteria that users consider in using a Web site. The 
other three are designed to put these criteria in some perspective by characterizing the 
user community’s information needs and search behavior generally and specifically their 
Internet and Web use. 

1 . What characteristics of Web sites affect use? 

2. What kinds of information related to their work activities does the user 
community need? 

3. How do they currently find this information, considering a wide range of sources, 
not just the Web? 

4. How do they use the Internet and the World Wide Web currently and how is this 
behavior likely to change as more resources become generally available or 
available in a form useful for their purposes? 

Methodology 

For this exploratory research, the project gathered data from business faculty with 
Internet experience through a questionnaire distributed electronically and a half-day focus 
group session held in an electronic environment. Approval to conduct research involving 
human subjects was received. A copy of the approval form is included in Appendix A. 

Participants. A total of 32 people were invited to participate in the focus group 
session in two stages through a modified snowball sampling approach. All contacts prior 
to the session, including the invitation, occurred via e-mail. The Invitation to Participants 
is included in Appendix B. 

Initially a sample of 5 participants was selected from faculty members from each 
of four graduate business schools in the Baltimore/Washington metropolitan area (N=20). 
The Business Schools included were: University of Maryland, Georgetown University, 
George Washington University, University of Baltimore. The initial group was 
selectively chosen to represent both genders, a range of fields, and teaching experience in 
graduate and undergraduate courses. At the second stage, faculty members who could not 
participate were asked to suggest colleagues, and other faculty were selected to insure 
representation from each business school. An additional 12 were identified and 
forwarded invitations. Excluding those in the original sample with schedule conflicts, but 
including new invitees, the modified sample was 24. From this group, nine (38 percent 
of the modified sample) participated, four of whom had been suggested by their 
colleagues. 
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The nine participants represented the four different business programs. Three 
were women, six were men. All indicated that they taught at the master’s level, which is 
not surprising considering the emphasis on the MBA degree in these programs; six 
indicated they taught also doctoral students; and four indicated they taught at the 
undergraduate level. The fields represented included accounting, finance, human 
resources management, marketing, and strategic planning. They all had doctoral degrees. 

As a group the participants were long-term, frequent e-mail users with four or 
more years of almost daily usage. Their e-mail usage had paved the way for their Web 
use, which had begun more recently. For most, their Web experience had begun within 
the last one or two years, but they generally used it several times a week to several times 
a month; only one person indicated very infrequent use. 

Questionnaire. A brief questionnaire sent to each participant prior to the focus 
group session provided background information on the participants’ computer and World 
Wide Web use so that questions could be modified to reflect the range of 
experiences/expertise among the group. The response rate was 100 percent. The e-mail 
questionnaire is included in Appendix C. 

Focus Group Session. The focus group session itself was held in the AT&T 
Teaching Theater at the University of Maryland, using the VisionQuest software. A 
description of the software is included in Appendix D. This software allows for a range 
of collaborative tasks, including brainwriting, commenting, grouping/sorting lists, and 
rating items on a list. In this session, two researchers knowledgeable about both business 
information and Web capabilities moderated discussions between the computer tasks, and 
a computer facilitator skilled in the use of the software occasionally asked for 
clarification of terms/ideas. The aggregated data were available in electronic form 
immediately after the session for the researchers. In general, the statistics provided by the 
software package are not very sophisticated but are appropriate for a focus group session. 

The focus group session consisted of two subsessions. The first emphasized 
general business information gathering behavior, including the use of formal (print and 
electronic) and informal sources, e.g. personal contacts, which could have occurred via a 
range of media, including e-mail. The second elicited specific data related to use of the 
Web. Table I shows the questions and/or tasks covered in the individual subsessions. 

During the session, each participant entered his data in the form of ideas, 
rankings, or ratings via a computer. Periodically he submitted the data to the server, 
which processed the data to provide aggregated findings to facilitate group discussion. 

No data were ever identified with an individual. Occasionally, when the data were used 
for a subsequent task, they were submitted to group scrutiny before that task to clarify 
ambiguities and remove duplicates. 
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Table I. Questions/Tasks Covered During the Focus Group Session 



Question 


SoftwareTask* 


Session I. Use of Business Information 




What types of business information do you use in connection with 
your work as a professor? 


Brainwriting 


Please estimate the importance of each of these types for you 
personally. 


Rating 


For each type of business information, please list the sources you use 
to obtain the information. 


Commenting 


For each type of business information - if it were on the Web now, 
would you use it? 


Categorizing 


For each type of business information - if it were on the Web now, 
would you have your students use it? 

Session II Use of Web Information 


Categorizing 


Consider the Web sites you have found most useful for business 
information. What features do you like about these sites? 


Brainwriting 



For each feature, please rate its influence on your use of a Web site. Rating 
What features have you disliked about Web sites that you have tried to Brainwriting 



use to obtain business information? 

For each feature, please rate the negative influence of the feature on 
your use of a Web site. 


Rating 


If business information were available from multiple sources, what 
would prompt you to choose the Web over other sources? 


Brainwriting 


How do you find out about useful Web resources? 


Brainwriting 


What factors limit your use of the Web for business information? 


Brainwriting 


Please rank the sources of information about Web sites in order of the 
frequency with which you use them. 


Rating 


What services or information would you be willing to pay for on the 
Internet? Consider personal, departmental, or grant payment. 


Brainwriting 



Please list names and/or descriptions of any Web sites you particularly Brainwriting 
like. 



* Refers to the specific software task in VisionQuest software. 
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Analysis 

Criteria 

Through “brainwriting,” which is designed to stimulate and elicit ideas, the focus 
group identified 46 positive features and 43 negative features of Web sites. They 
subsequently rated these on a 5-point scale on their degree of influence on subsequent use 
or non-use of Web sites. 1 The features were screened to merge duplicates and to remove 
system-related features and ambiguous or vague features. The adjusted totals were 33 
positive features and 18 negative features (72 percent and 42 percent of the original 
respective lists). The remaining site features were clustered based on similarity of 
interest, disregarding the positive or negative aspect of the criterion. Items relating to 
more than one cluster were duplicated in each cluster. Table II shows examples of coding 
for the relationship between an influential feature and its corresponding cluster. 

The items were then ranked within the positive and negative groups according to 
the average ranking; the ranks for positive ranged from 3.1 to 4.9, for negative from 2.7 
to 4.3. The median rank was used to determine the most influential factors (3.56 for 
positive factors; 3.44 for negative factors). Table III shows these factors. The positive 
features to the left promoted use; the negative factors on the right contributed strongly to 
discontinuing use. Six clusters emerged in the analysis: appearance, content, linkage, 
special feature, structure, use. The ranks for the items included in each cluster were 
combined and averaged. For positive factors, all clusters appeared in order: use (4.4); 
content (4.3); linkage (4.1); structure (4.1); special feature (3.7); appearance (3.6). It is 
apparent that the participants rated use and content above appearance. Structure and 
search capability would generally contribute to use. For negative factors, only four 
clusters appear. In order, they are: use (4.3); content (4.0); structure (3.7); and linkage 
(3.6). 



In addition to Web page/site features, the focus group members reacted positively 
or negatively to factors extrinsic to the Web sites. Usually they were related to search 
system features or to Internet or World Wide Web features. Slow response times and 
system crashes, for example, were strong negative influences; the availability of search 
engines and the flexibility of access to the Internet, both in terms of time and location, 
were strong positive influences. In terms of search engines, they commented negatively, 
for example, about inefficient searches or misinterpretations of search query words and 
retrieval of extraneous information. 



General Information-Related Behavior 



The participants identified and ranked 49 different types of information they used 
in their work activities, which included teaching, research, and consulting. The 
participants ranked the importance of the types of information on a 5-point scale, with 1 
as “not at all important” and 5 as “very important.” Users had some difficulty in 
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Table II. Sample of Relationships Between Features and Clusters 



Feature 


Cluster 


Influential Negative Features 

Slow response time 

Waiting for links and connections 

Search engines that take forever to give useful results 

Systems crashes from poor software, system overloads 

Getting lost 


System 

System 

System 

System 

Navigation 


Superficial information; not informative 
Sites that go nowhere say nothing 
Advertisements 


Content 

Content 

Content 


Wasted search time 
Inefficient search 

Search engines that do not really search or take forever to 
give useful results 


Search 

Search 

Search 


Influential Positive Features 




Usefulness of information 
Currency 


Content 

Content 


Linkage to relevant sites 


Linkage, content 


Ease of use 

Ease of navigation through a database 


Use 

Use 


Intelligible structure 
Well-organized 

Ability to get overview of structure 
Straightforward 


Structure 
Structure 
Structure, Use 
Structure 


Reasonable response time 


System 
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Table III. Web Page Features Considered Most Influential in Decisions to Use 



Positive Features 


Negative Features 


Use 


Ease of use 


Navigation problems 


Ease of navigation through a verbal database 




Ability to get overview of structure 




Content 


Usefulness of information 


Superficial information 


Currency of information 


Uninformative content 


Concise, non-repetitive information 


Advertisements 


Contains information not easily or readily 


Repetitious content 


found in library collections 


Boring text 




Lack of currency of information 


Structure 


Well-organized 


Long text 


Intelligible structure 


Poorly named subsections 


Ability to get overview of structure 




Straightforward 




Breaking up of text 




Innovative presentation and organization 




Linkage 


Linkage to sites with relevant information 


Linkage that goes nowhere 


Ability to switch back and forth between topics 




Special Feature 


Predictable address 




Search capability on single page 




Appearance 


Visually attractive 




Printable without overly-darkened areas 





Note: Broad categories are arranged according to ranked order, which was the same for both 
positive and negative factors. 
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differentiating between information and sources. Many of the items referred to specific 
sources or means of conveying information, not to types of information. 

The large set was screened to remove duplicates, ambiguous items, curriculum or 
course-related items, such as student feedback, Web-related items, and items referring to 
specific sources, not types of information. Sometimes a highly specific particular source 
was translated into a type of information. Twenty-nine of the items remained (59 
percent). The average rank for these items was 2.8 (S.D. .7), but ranged from 1.9 to 4.4. 
These items were clustered into broad categories, and the average rank of items in each 
cluster was computed to serve as a basis for ranking the broad categories. 

Table IV shows the broad categories, according to their ranked importance: 
literature (3.9); case studies and other stories (3.0); government information (2.8); news 
related to business-related events (2.7); directory information (2.7); statistical or 
numerical data (2.4); and standards and procedures (2.1). The more specific types 
mentioned by the respondents provide the examples noted for each broad type. 



Table IV. Types of Business Information Used by Participants 



Literature 

Includes scholarly articles in business or professional journals. 

Case Studies and Other Stories 

Includes formal case studies likely to be used in class assignments, anecdotal 
information about business decisions by major participants. 

Government Information 

Includes government procedures, legislation, regulations. 

News Related to Business-related Events 

Includes both current articles and news summaries appearing in the business or 
general news press. 

Directory Information 

Includes current addresses, telephone numbers, lists of boards of directors. 
Statistical or Numerical Data 

Includes historical, current, and forecast data about elements such as stock prices, 
balance sheet data, profitability statements. 

Standards and Procedures 

Includes, for example, accounting principles from the official professional group or 
“best practices.” 

Note: Information types are presented in order of importance to focus group members. 
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The types of information and sources used to locate those types are subject or 
discipline specific and the relevant level is probably not “business” but subfields of 
business, such as “accounting,” “finance,” and so on. Generalizing about types of 
information or sources used to locate them, as noted in the next paragraphs, is therefore 
questionable with such a small sample. In part, these problems justify grouping specific 
items into broad categories or noting the information only as part of case studies. At the 
broad category level, the identification of types of information is probably more reliable 
than the rankings attributed to them by this focus group. Not surprisingly, the categories 
ranked highest are those which are applicable across a broad range of business fields. 

Not all fields relate to standards or statistical information. 

The participants also indicated the sources they currently used to locate the most 
heavily ranked types of information. In general, they used a variety of interpersonal, 
print, video, and electronic sources, with multiple types of sources often used for each 
category. The following examples serve to illustrate their current, eclectic approaches to 
finding useful information. 

• For both professional and scholarly literature, they mentioned specifically 12 
magazines and newspapers, including, for example, the Wall Street Journal, the 
Washington Post, Business Week, Fortune, Journal of Finance, and Wired. Their 
references to the online versions of several of these titles indicates some 
acceptance already of full-text of journals or newspapers in electronic format. 
They searched two electronic databases to locate these articles: ABI/Inform, a 
broadly-based business literature database, and PsychLit, a comprehensive 
psychological literature database, which covers the psychological literature useful 
in human resource management or marketing research. Neither of these cover the 
newspapers. They also mentioned an Internet search engine and an online public 
access catalog. 

• For case studies, which are widely used in classes, the participants relied on 
published case studies, such as those published by the Harvard Business School 
(HBS) and the Darden School of Business or included in commercially published 
casebooks. In addition, they consulted electronic sources, e.g. colleagues’ Web 
sites; videos of company practices/issues; sometimes they wrote their own. 

Several participants commented favorably on being able to preview HBS case 
studies by downloading them from their Web site. 

• For war stories about experiences, the participants drew on their own personal 
experiences, books and journals, discussions with trusted professionals, 
newspaper articles, conference events, and guest speakers. 

• For practical applications of business practices, they used solely interpersonal 
sources, gleaned from interactions with professionals in the community and in 
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professional organizations, continuous networking with practitioners, and 

attendance at professional development meetings. 

It is apparent in these brief case studies of locating useful information that the 
participants are already adapting information gathering patterns based on an pre- 
Intemet/Web environment that drew heavily on personal contacts, both within their work 
setting and through conferences and connections with the business community, and 
printed sources, some identified through formal means such as checking indexes and 
some which were monitored as part of their current awareness efforts to read current 
issues of relevant journals and newspapers, to the new world of electronic access, both to 
colleagues and other personal contacts and to printed sources. 

Web Use 

How is behavior likely to change as the participants become more aware or 
knowledgeable about Web resources and as more business information relevant to their 
purposes becomes available? The basic categories listed in the first two columns in Table 
V are derived from business information the faculty indicated they were already using in 
connection with teaching, research, or consulting (Summarized in Table IV). Table V 
shows their reactions about use if the information were available on the Web. 2 The 
asterisked items in the second and third columns show the overlap with the broadest set 
of participant choices in the first column. 

The only new option the participants would recommend to students they would 
not necessarily use themselves is foreign business journals; these schools have many 
foreign students in their programs. The participants generally would use more 
information types than they would be willing to pay for. For Column 3, the participants 
generated the types of information anew, specifically in terms of willingness to pay for 
them. Their responses reflect the current market status for most of these types of 
information, and there is some overlap with types already mentioned in Column 1 , but 
there are some notable differences. In the first column, they anticipate getting company- 
related information by going to individual corporate reports or corporate Web sites and 
would probably be using this for individual company or small groups of companies 
analyses. Many companies have Web sites, and the faculty undoubtedly foresee 
provision of this information by publicly-held companies as a public service companies 
should provide free of charge. On the other hand, they would be willing to pay for a 
database which compiled certain kinds of data, e.g. stock prices, for many companies to 
allow them to do cross-company analyses easily. Two of the information types are 
somewhat novel -- the current subject-specific phone/address lists, and the “inside 
information” from prominent individuals. Presumably the prominent individuals would 
be people involved in business deals or noteworthy observers, but the kinds of individuals 
were not identified. The participants show, for example, a willingness to pay for expert 
opinion in the case of the market analysts’ reports. 
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Table V. Reactions to Web Availability of Business Information 



Willing to Use Willing to Refer to Students Willing to Pay For 



Best practices 
Case studies 

Corporate annual reports, 
related information 
Databases 

Journal articles showing 
company practices, problems, 
issues 

New developments in 
information technology 
Practical applications of 
business practices by 
professionals 

Professional journal articles 
Professional organization 
information, e.g. standards 
Research journal articles 
Wall Street Journal or other 
business news sources 
War stories 
Course syllabi 



* Corporate annual reports, 
related information 

* Databases 

Foreign business magazines 

* Journal articles showing 
company practices, problems, 
issues 

News summaries 

* Practical applications of 
business practices by 
professionals 

* Professional journal articles 

* Professional organization 
information, e.g. standards 

* Research journal articles 

* War stories 

* Course syllabi 



* Current price and financial 
information for companies 

* Databases 

* Electronic versions of 
journals 

Inside knowledge of 
prominent people 
Market analyst reports 
Topic-area lists of current, 
relevant addresses 

* Wall Street Journal or other 
business news sources 



Notes: The first and second columns list any information type mentioned by six or more 
participants. The third column lists all information types mentioned. 

* Indicates categories duplicated in Column 1. 



Factors which would prompt the participants to use business information on the 
Web rather than in other media or formats are: 

• Cost, 

• Convenience, both in terms of time and physical access , 

• Ability to get electronic or paper copies easily, 

• Characteristics of information, such as the credibility of the source, 

• Ability to rely on themselves rather than an intermediary, 

• Response time, 

• Environmental concerns, and 

• Ease of use, which could refer to many of these factors in itself. 

To find resources located on the Web, they relied in order on: Web-related 
features, especially search engines and links; colleagues, including students; print 
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sources; and formal training by librarians or other information professionals. For 
example, related to colleagues, they mentioned exchanging bookmarks and seeing 
descriptions of good sites on subject-related listservs. For print sources, they noted, for 
example, lists of Web sites in professional newsletters, URLs included in advertisements, 
newspaper and journal articles, and recent textbooks. One participant mentioned a 
specific guide to Internet resources for her field. In the vocal discussions, they expressed 
some misgivings about their abilities in finding information that could be of use to them 
and were interested in any tactics they might adopt to help them to locate relevant 
information. 

For inhibiting factors to Web use, the respondents mentioned: 

• Self-related problems, e.g. ignorance, inexperience, lack of time, tendency to use 
familiar sources, need to balance student’s over-reliance on the Web, and 
competing demands, • 

• Web-related problems, e.g. site fragmentation, lack of filtering, more raw data 
than analysis, and 

• System-related problems, e.g. inability to use anything but a text-based system 
from home, slow response time on the university’s network, software lockups. 

Two findings about inhibiting factors are notable: first, many of the self-related problems 
are cognitive or psychological; and second, price is not mentioned as a inhibiting factor. 



Part III. The User-based Design Process and Its Application 

As mentioned earlier, the project team developed and implemented a user-based 
design process. In the user-based design process, the user occupies a central place during 
the design process. The Web site is being designed for the user, and his input is solicited 
throughout the process. Whenever possible design decisions reflect the user’s rather than 
the designer’s viewpoint. In the early stages of developing sites on the Web, Web 
page/site designers relegated users to a secondary position because, as a group, users’ 
experiences were limited and idiosyncratic. But, as the Web matures, a body of users is 
developing that uses the Web not only intensively but also extensively so their experience 
can be tapped at various times during the design process. They have a good sense of the 
site features that influence their use of a Web site and can participate effectively as 
partners in the design of Web sites to suit their particular information needs. 

In the user-based design process followed in this project, user input is solicited at 
several stages: 

• early in the project to determine the criteria users apply to the Web sites they use. 
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• after a preliminary design of the Web site to elicit feedback and comments or to 

evaluate certain aspects of the site. 

• when the Web site is operational to elicit continuous (or continual) feedback and 

suggestions for additions and/or modifications to the site. 

Figure 1 shows four stages in the user-based design process. The stages follow a 
chronological sequence and move from the bottom up to reinforce that each builds on th 
other. The key element that distinguishes this process is that users are consulted at each 
stage of the design process, not simply at the end, after design and content features have 
been largely determined by the design team. 

As noted in Figure 1, for the user-based design process to work effectively, the 
design team should have a clear understanding of the usual task-related information 
seeking and use behavior of the client group. This background provides a context for 
understanding the users’ specific comments about Web use. Ideally the designers should 
understand the kinds of information the users typically use, perhaps some specific 
sources, what form those sources are in (print, video, etc.; electronic, hardcopy), how the 
users locate them, and what problems they experience in looking for information they 
need. This picture should include not only use of formal, print sources, such as a journal, 
but also informal contacts with colleagues or others, such as businessmen or students. 

For a Web resource to be effective in responding to information needs, the designers 
should have a clear understanding of those needs and of the range of competing and 
complementary sources the users are accustomed to consulting. 

This background information is placed before the stages in the design process in 
Figure 1 for several reasons. First, all of the decisions during the design process occur 
within this context; it should influence and enlighten the data-gathering, analysis, design, 
and implementation decisions. Second, in some cases, this information can be obtained 
from existing studies of user information-related behavior, so this part of the project 
actually occurs as part of an organized review of relevant research before contact with the 
user group. Unfortunately, use studies of the relevant user groups do not always exist. 
Another failing of these use studies even if they do exist is that their reliability and 
validity usually decreases with time since the information scene changes so rapidly. In 
this project, for example, useful studies existed, but they were done before widespread 
use of the Internet and, as a result, did not provide a good picture of electronic 
information use in the context of total information use. As a result. Stage 1 's information 
gathering expanded to include getting information about total task-related information 
use behavior in addition to criteria-oriented information. 

Information-Gathering Stage 

Determining the criteria the client applies in judging the usefulness of a Web site 
is the major task at Stage 1 . The data can be gathered through a variety of means, such as 
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questionnaires or interviews. As noted in Part II of this report, the most important data- 
gathering for this project occurred during the half-day focus group session with business 
school faculty. The data provided rich insights into how business faculty are using the 
World Wide Web and the criteria they use in judging sites. 

The audience for the test site was the same as the one reflected by the focus group 
participants, business school faculty. Some suggestions for the content were drawn from 
the focus group results. This information was used in combination with the objectives 
and the design criteria to make a variety of decisions during site development. 

Development Stage 

Several critical tasks occur at this stage. Obviously the designers need to 
understand the criteria since they will serve as the underlying basis for the design. To do 
this they need to operationalize the criteria and assess their implications for design. In the 
process, they also rank the criteria based on feasibility. Then, the designers translate the 
criteria into Web site features and design a preliminary version of the Web site. Since the 
overall approach is to allow for continual user feedback, at this stage the designers 
consciously have to incorporate feedback mechanisms into the design. 

Operational definitions were established for the criteria in this study on the basis 
of the comments made by the users. It is important that the criteria be operationalized 
from the perspective of the user and not the designer. These criteria can be used to 
generate a wealth of specific suggestions for site design as shown in the following 
paragraphs. Table VI contains the operational definitions for the criteria. 

Content Content was the highest rated of the criteria identified and, as is shown 
in Figure 2, is core to the design process. All other criteria impact the content of the site. 
Clearly, if the content requirements are not met, users in this group will not use a Web 
site. Emphasis here is on useful and current information. To understand what is useful, it 
is necessary to have an understanding of the information seeking behavior of the group. 

In addition to useful, an element of novelty is desired, that is, inf ormation that is not 
easily accessible elsewhere. This finding is extremely important for Web designers. 

Users will visit a site only if it offers them useful content. The importance of content is 
stressed by Tufte (1997) in his discussion of information design and computer interfaces. 

The brainstorming session on content went beyond the development of this 
general content definition and provided specific examples of resources that permitted the 
development of categories of content that would be useful for the development of the 
Web site. For example, in mentioning financial data for companies, the focus group 
referred specifically to filings with the Securities and Exchange Commission or to Value 
Line analyses. Referring to business-related news, they mentioned Wall Street Journal 
and the Bloomberg News Service. 
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Table VI. Operational Definitions of User Criteria 



Use The site is easy to use. An overview of the site and appropriate navigation structures 

are available. Users do not get lost easily. 

Content Useful information. Current information. Concise non-repetitive information. 

Information not easily or readily found in library collections. Absence of the 
following: superficial and repetitious information, uninformative content, 
advertisements, boring text, lack of currency. 

Structure The site displays an intelligible, straightforward organizing scheme. Text is broken 

into appropriate, well-labeled subsections. Large blocks of text are minimized. 

Linkage Pages provide links that integrate relevant information at the site and at other sites. 

Link provide access to related topics allowing serendipitous discovery of 
information. All links function; broken and under construction links are avoided. 

Search Search support for page and site searching are provided. Searching produces a 

precise list of helpful sites or pages with a minimum of processing time. 

Appearance The site is visually attractive on-screen. Any given page contains few graphics and 
these are appropriate to page content. Graphics are not essential to site use; if 
graphics are turned off or a text-only client is used, the site remains fully functional. 
Pages result in attractive printouts without large dark areas. 

Note: The criteria are arranged in ranked order of importance. 



The examples of specific sources mentioned above bring to light an interesting 
point relating to the importance of authoritative information to users. During the 
brainstorming session, the word "authority" did not surface. Yet, the importance of 
authoritative information was obvious in the sources cited by the users. It seems that 
authority is intertwined with content and is implicit. That is, when a user states that 
information must be useful, they are referring not only to topic coverage but also to the 
source or producer of the information. So, the users most often provided qualified 
examples of content. Rather than indicate a need for a general type of information, such 
as case studies, the users specified specific authoritative sources for this type of 
information, such as the Harvard Business School. Implicit qualifying by authority 
extended across all types of information. 

The users' discussion of desired content largely structured content development of 
the test site. All of the main resource categories visible on the home page (Figure 3) were 
discussed in one form or another as providing useful content to focus group members. 

The featured site category was directly suggested by users' discussion of the value of 
discovery of novel information resources. Initially the information sources provided are 
constrained to content provided by existing sites with value added largely by annotation, 
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Figure 2. User-based Criteria and Their Relationships 




Note: Because this is a uni-dimensional display, not all relationships are evident. 
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organization, and linkage. However, mechanism are incorporated into the test site for 
users to suggest needed content. Currency of update is indicated on all site pages and a 
regular schedule of link checking is followed. 

Structure. The structure criterion suggests that arrangement of site content 
should not be incidental or arbitrary. Users expect not just a clear structure, but one 
which makes sense. The home page or major navigation pages should indicate the 
structure of the site's content. When categories organize the site structure, it should be 
easy for users to distinguish between categories. Users prefer content to be broken into 
small chunks. This suggests that multiple short pages are easier to manipulate and 
comprehend than long pages requiring scrolling. Chunks of text could be organized in a 
variety of ways including linear structures, hierarchies, or networks. The concern for 
intelligibility indicates that labeling must be sensible, consistent, unambiguous, and 
jargon-free. 

In the design of the test site, seven categories were created at the top level and are 
displayed on the home page (See Figure 3). The categories were drawn from users 
suggestions and to the extent possible borrowed their terminology. The categories are 
distinct from one another and are set off from supporting elements such as logos, revision 
dates, and access to evaluation forms. Content is organized at subsidiary levels in the 
hierarchy as well. For instance, academic business listserv lists are grouped by subject. 
Publications are organized by format. 



Figure 3. Original Top Level Page 
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Linkage. Although linkage relates to structure and organization, it emphasizes 
the need for integration of information between and within sites. Links should pull 
together related information to allow the users to explore and discover information 
serendipitously. Linkages are the results of intellectual effort by site designers to identify 
and provide, connections to related information available at other sites and thus to add to 
the perceived value of the original site. The inclusion of links that are "under 
construction", incorrectly coded, or outdated can quickly convince users that a site is a 
waste of their time and resources. Notices of most recent update or verification provide 
users with some assurance that links are reasonably current. 

At the test site, links integrate information within the site and bring together 
information from across the Web. Internal links provide access to information justifying 
decisions about the inclusion of content. They also provide access to opportunities to 
evaluate the site or suggest content. External links are the main value added by the 
service. A variety of business sources are brought together and integrated by content, 
format, and publisher. 

Use. Clearly, structure is related to ease of use. The content indicators and 
organizing structures should be integrated into the navigational system. Links and- 
buttons facilitate navigation in a Web environment. Overviews make a site easier to 
understand and easier to use. While many think of home pages as providing the main site 
overview, intermediate overviews at alternate levels can also be helpful. If site structure 
is even crudely hierarchical, top-down navigation is easily facilitated by making 
subsection names into links connecting an overview to the subsections. Bottom-up 
navigation is not as easily facilitated by simple links, but many sites use button bars 
effectively to provide a summary of the site organization and link subsections to each 
other and to overviews. Small page size can also facilitate rapid navigation by reducing 
the need for scrolling. 

The test site uses links and buttons to support top-down navigation. Tables are 
used to facilitate page organization and minimize white space and consequently the need 
for scrolling. The organizing structure links directly to subsections. Shortcuts to popular 
subsections are provided by listing two hierarchical levels for many of the main 
organizing categories. For instance, "News" is linked to a subsection organizing 
newspapers and magazines, but links directly to "Newspapers" and "Magazines" are also 
available directly from the home page. 

Search. Search capabilities should perhaps be an obvious part of site design, but 
implementation is rare since the development of a separate search module is required for 
site searching. Sadly, non-functional search features or search features that function quite 
poorly are common among Web sites. Users want responsive, effective searching, and 
this area presents the greatest challenge to current designers, particularly those with 
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limited resources. Currently the test site does not support a site search capability. This 
functionality is high on the priority list for future enhancements on the test site. 

Appearance. Although users are clearly aware of site appearance, it is apparent 
that they regard it as cosmetic and mentioned many hindrances to use created in the 
search for an attractive site appearance. Unattractive graphics, excessively numerous 
graphics, over-large graphics, dark graphics, and graphics that are required for effective 
site use are all deplorable. Although advice to system designers almost un anim ously 
discourages spurious use of graphic elements, clearly many sites are not implementing 
this advice, perhaps because the degree of offense to users can be easily overcome by 
other site virtues. 

In the test site, tables create an attractive layout of page elements and impose a 
uniformity of appearance on pages. A few small graphics enhance consistency of 
appearance between pages; these double as buttons to provide navigational support. A 
white background was used to improve readability. Other than this background 
alteration, basic defaults for text appearance were not altered. 

With the exception of search functionality, all of the user-generated criteria for 
use provided significant guidance for the initial test site design. The criteria were easily 
operationalized with the operational definitions suggesting both what should be available 
in terms of content and how the content should be presented to users. The implications 
ranged from relatively superficial issues about appearance and terminology to more 
profound issues such as site structure, category definition, and selection policies. 
Ultimately, the test site content was organized into eight major sections, each suggested 
by comments from focus group participants. Content emphasizes resource discovery and 
ranges among directory type information, links to resources external to the site, and 
instructions and tutorials. 

The organization of the site is indicated on the home page and reflects 
terminology natural groupings used by business faculty. The site functions largely as a 
resource to provide access via linkage to a wide range of Web-based information. As the 
criteria structured content and design decisions, they also provided a structure for initial 
design of feedback mechanisms. The criteria categories are evident in the preliminary 
design of the site's evaluation form (Figure 4). This form is not simply an open request 
for feedback but demonstrates a serious interest in user's thoughts and provides a 
framework to encourage users to provide information at a level of detail and consistency 
that allows site designers to use feedback as a basis for informed decision making. The 
evaluation form is an integral part of the site design rather than a temporary adjunct and is 
expected to provide ongoing feedback for site development. In addition to the overall 
evaluation form, forms are also available on most pages encouraging users to suggest 
resources for addition to the site's content. 
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Figure 4. Evaluation Form 



■U HIW— H1IOIBIH1IM W11MW— 

aJ 

£»e £dit View go 



SBS 



Bookmarks Options Directory Wlndo> 



Jbldp 



tion: J http: //www! warn. umd. edu 7 — khohn/ hit miy disclosure^ orms/a vol. hitm 



fiiEg 






mm 






Evaluate the "Tools for Academics" Web site 

m m J7o >4 curr«nf/y non^l mctibno/. ** ** 

How often have youuaedthe aite 

O Once O 2-3 times O More than 3 time* 
Current affiliation 

O Faculty’ O Student O Business professional 
O Other(Please describe) J 



Please describe the types of business information you use 



What do you like best about this site*? 






What do you like least about this site? 



Please rate the following features 
The site is easy to use 

O Strongly agree 

O Agree 
O N ot sure 
O Disagree 
O Strongly disagree 
Tile information is us efuJ to me 
O Strongly agree 
O Agree 
O Hot sure 
O Disagree 
O Strongly disagree 
Tlie organization of tlie site is effective 
O Strongly agree 
O Agree 
O Hot sure 
O Die agre e 
O Strongly disagree 
Tlie pages are attractive 

O Strongly agree 
O Agree 
O Hot sure 
O Die agre e 
O Strongly disagree 

Tlie amount of information on individual pages is about right 
O Strongly agree 
O Agree 
O Hot sure 
O Disagree 
O Strongly disagree 

The site helps me discover new information resources 
O Strongly agree 
O Agree 
O Hot sure 
O Die agre e 
O Strongly disagree 
A dditional comments 





| ft ub irnit Q u crry ■: Ifias ofrij 



Alpha product version created under contract to Disclosure. Inc. 



l^ocumentDone 









O 

ERIC 



30 BEST COPY AVAILABLE 










25 



In some instances, the evaluation stage may also include testing. Usability testing 
may be appropriate in those cases where specific design features are used to accomplish 
particular tasks. For instance, classic usability testing might be applied to determine the 
effectiveness of a site's search function, to test a site's effectiveness for resource 
discovery, or to evaluate some special functionality of the site. 

Evaluation Stage 

Evaluation of the Web site is a stage in and of itself in the user-based design 
process when the site is in its initial development, and it is also included as a part of the 
implementation stage on a continual basis. Before the test site becomes operational, it is 
imperative to have users evaluate the site in order to determine if the criteria were 
translated appropriately into design. It is important to obtain feedback on the design 
criteria. This can be accomplished through questionnaires administered to the user group 
who are asked to use the site and respond to specific questions. Another option is to use 
the talk aloud protocol to illicit comments from users with a researcher present. Both 
methodologies result in data useful for modifying the Web site. This process is iterative 
so that evaluative comments are solicited from the user group after changes are made 
based on previous suggestions. The process continues until reactions received from the 
users indicate that the site is ready for implementation. In some instances, the evaluation 
stage may also include testing. Usability testing may be appropriate in those cases where 
specific design features are used to accomplish specific tasks. For instance, refining the 
design of a Web-based currency exchange calculator or yellow page directory are tasks 
that can be enhanced by usability testing. In contrast, browsing and information 
exploration are quite open-ended and unspecified. 

Two groups evaluated the site: the focus group members, and a larger selected 
group of academic users contacted by representatives of the funding agency. Focus 
groups members were sent an evaluation form via electronic mail. The e-mail evaluation 
form is included in Appendix E. The latter group were contacted informally by firm 
representatives and reported back to the development team verbally. Including the latter 
group in effect tests the generalizability of the criteria and design to a broader 
community. The feedback received from the two groups is being analyzed and 
considered within two external constraints. First, the comments must fit within the 
objectives of the site, and second, they must be feasible to implement. Feasibility is 
based on economic and technical considerations. Any suggestions which are not 
incorporated into the Web site at this time will be saved and reviewed periodically to 
determine if they can be incorporated or are no longer useful at a later date. 

Evaluation is iterative at this stage. If the modifications to the Web site based on 
their comments are substantial, the two groups will once again be asked to react to the 
suggestions. And the process will continue until an acceptable version of the Web site is 



ERIC 



31 



26 



ready. The final version of the Web site is described in the implementation section of this 
technical report- 

Implementation Stage 

At some point in the user-based design process, the site is considered complete 
and finished. The final version of the Web site represents not only the design as based on 
the user group’s initial criteria but also their feedback and possible feedback from other 
members of the same community to preliminary versions of the site. No Web site 
developed through this user-based design process is ever considered “finished,” since 
user feedback and suggestions occur continually. This continual feedback should 
continue to be considered in making decisions about site design. But design is no longer 
the major focus of the site developers; implementation and maintenance become 
paramount. 

In the implementation stage, the site becomes operational and available. Real 
users begin to incorporate the site into their normal information-gathering behavior. At 
this point, site developers may find it useful to announce the Web site’s availability 
formally to members of the primary user groups, using both electronic and non-electronic 
means. Otherwise, the site’s use is likely to occur through references from Websearch 
services and word-of-mouth. 

The formal feedback mechanisms incorporated into the Web site at Stage 2 are 
activated. Usually these are likely to consist of evaluation and/or suggestion forms. The 
evaluation form developed for this site asks users to comment on the site on the basis of 
the design criteria: ease of use; usefulness of information; organization of site; 
attractiveness of pages; amount of information on pages; and usefulness in discovering 
new information. Users are also asked to identify what it is that they like best or least. 

The evaluation form is included in Figure 4. In addition to evaluating the site, users are 
asked to suggest new business resources to add to the site. The suggestion form is 
included in Figure 5. 

Site developers continue to monitor the feedback and periodically assess the 
comments and suggestions. In addition, they note the use of specific segments of the site 
as indicated through the log. At this stage, a program should be established for continued 
development, which incorporates periodic emphases on design modifications in response 
to actual use, user feedback, and other changes in the information environment, such as 
the availability of new information. It is important to realize that user needs may shift 
over time and thus necessitate modifying the Web site. 

Aside from the information gathered during the research phase of this project, 
several other factors influenced development of the Web site. These are comparable to 
any considerations a design team may establish with the agency funding the site’s 
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Figure 5. Suggestion Form 



c Netscape -[Disclosure; Alpha Product Suggest a business resource] 



••0c fiookmaikt . 



BQu 






Suggest a business resource 

**T7dsform is currently nonfunctional. ** 
Site URL: 



U 



*1 «•* * 






Site name: 



U 



g .« fi ?»■ sc .as* •« •»' «r 



;iK'_ ■R.^-€«^-5S ;& ; Ci it* 



Site desorption: 

Please tell us what the site offers that you think would be of interest to the aca 
business community. 



H 



Your email address: 



k?, .1 'if ff a ■& 31 V k i'/.-M 






il 

3 ? 



BEST COPY AVAILABLE 




33 




28 



development. Various meetings were held between the project team and Disclosure staff. 
In those meetings, selection criteria and development guidelines were established. The 
Web site was transferred to Disclosure in early December 1997, and the integration 
process began at that time. A complete chronology of the project is found in Appendix F. 

The actual development of the site followed in large part the criteria developed 
during the research phase with a few minor modifications. For the site, preference was 
given to the provision of free information. Preference was also given to electronic 
resources, but pointers to print information or to fee-based print or electronic information 
were included if an identified content need could not be met with free electronic sources. 
Both the organization and researchers agreed that the site would not be a self-serving 
commercial site designed primarily to promote the services provided by the funding 
agency. The only constraint placed on the development by the funding agency related to 
the financial section of the Web site. The development of this section was left primarily 
to the funding agency. The organization is committed to providing and maintaining a 
Web site with a broader objective of linking the academic business community to useful 
electronic and print resources. 

Both parties agreed to develop a site that would allow for supporting the 
development of new Web-based information resources. In the long run, some new 
resources may be developed by this firm and others. They may be free or fee-based. This 
mix of free/fee-based services already exists for electronic business information. In this 
early period, both researchers and funding organization agreed to emphasize initially the 
free resources, based on knowledge of existing patterns of use of electronic information 
by academic users generally. 

Content areas were added to the site during its development based on suggestions 
made by the development team and requests from the funding agency. The user group 
was broadened to include the general academic business community: students, faculty, 
and business librarians. Ideally, members of these other segments of the community 
should also have been surveyed to identify the content and design factors they consider 
appropriate. This was not done because of time and funding constraints. But the content 
was expanded, and the entire site was re-organized slightly to accommodate the new 
content. Table VII shows the relationship between the new categories in the final design 
and those used in the original design The final top level page is shown in Figure 6. 

Part IV. Conclusion 

In existing design approaches, user input, where it occurs, is usually relegated to 
usability testing of specific design features. Methodologies have not been developed for 
utilizing user criteria as the basis for Web site design. This project takes the position that 
user input can and should be incorporated at an earlier stage in the design process. There 
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Table VII. Relationship Between Final and Original Categories 



Final Category 


Original Category 


Business Literature: 

New broad category with following four 
subcategories 


None 


News: 

Subcategory with links to free and fee-based 
news services and business-related 
newspapers. 

Journals: 

Subcategory with links to free and fee-based 
business magazines and journals. 

Case Studies: 

Subcategory with links to free and fee-based 
case studies available electronically or in 
print. 

Books: 

Subcategory with links to business-related 
books and book reviews. 


News: 

Major category containing newspapers 
and magazines. 

Magazines: 

Subcategory under News. Enhanced 
magazines to include journals. 

Case Studies: 

Included print and electronic case 
studies. 

None 


* Financial Information: 

Broad category which includes stock quotes, 
annual reports, corporate financial and 
international market information. 


Same 


Government Sites: 

Broad category with links to federal and 
international governmental business-related sites. 


Same 


Business School Information: 

Broad category which includes rankings and 
listings of programs, syllabi, research resources 
(libraries) and directories. 


Business Schools: 

Single focused major category became a 
subcategory within a broader category. 


Professional Associations: 

Broad category which contains contact 
information and links to business-related 
professional associations. 


Same 


Jobs: 

New broad category with links to sites oriented to 
business school students. 


None 


Tutorials and Guides: 

New broad category oriented to students. 


None 



0 

ERIC 



* Not completed by development team. 



35 



30 



Figure 6. Final Top Level Page 
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is a growing body of experienced Webusers that can articulate to Web site designers what 
they like and dislike about current Web sites and what affects their usage patterns. 

The results of the focus group session indicates that focus group sessions of 
relatively homogeneous groups of users yield useful data for the design and development 
of Web sites. The categories of criteria identified in the focus group session need further 
testing in larger samples within the same user group community as well as with other 
groups of users to test their generalizability. 

The user-based design criteria development process developed and used in this 
research gathers input from users with a significant degree of Webuse experience. As 
Webusers have expanded not only in numbers but also in diversity of interests and 
information needs, the Webhas also become in total more universal in its resources and 
information providers. What distinguishes this period of its growth is an increasing 
clamor for filtering, organizing, and designing the resources so that a better match is 
made between the appropriate resources and the user communities for whom they are 
intended. Among the many users of the Webare increasingly knowledgeable and 
experienced user communities that can discern useful information and Web site features 
that impact their continued use of Web sites. 

Users are brought together to discuss their information use in general and criteria 
that influence their use of the Web. To elicit the widest possible range of responses, users 
identify both positive and negative criteria. Users also indicate the relative importance of 
the criteria they identify. The data gathered are not simply translated into specific criteria 
but are analyzed and synthesized to generate broad design principles. Interpretation of the 
criteria is grounded in a broader discussion of general information use and overall 
Webuse. The over-riding and guiding principle behind the design is the importance of 
content to the user. Without content, users will not return to a site. 

The multi-stage, user-based process of developing a Web site used for this project 
is entirely appropriate for designing Web sites at this period of the Web’s development. 

It probably could not have been implemented as effectively at an earlier period, except for 
certain groups who were long-term users of the Web, because it depends so heavily on 
the existence of a knowledgeable, experienced user group. The user and his or her 
information needs and criteria forjudging Web sites are paramount in the process; the 
designer’s role is to translate these into an effective Web site. 

Another important feature of this process is that evaluation and feedback are 
incorporated into the implementation phase so that the users continually have the ability 
to suggest new resources, to react to organization or links, or to make comments about 
the appearance of the Web. The extent to which the user group will make use of these 
feedback mechanisms is still unknown but, if the mechanisms work, the site designers 
essentially have continual feedback to facilitate maintaining the site and making design 
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modifications as necessary or useful. In this project, for example, the users emphasized 
they wanted their information to be current and that they disliked sites that contained 
links to nowhere. It seems highly likely that, faced with links to non-existing sources or 
dated information, they will react if it is easy for them to react and if they perceive that 
the site designers act on their feedback. 

Although the data about use had immediate implications in this project for the 
design of a particular site, the process and results raise several intriguing questions. To 
what extent are the criteria that were identified in Stage 1 universal criteria? If they are, 
do all user groups rank them in the order identified in this project, with content and 
structure being more important than appearance? What characterizes information that 
clients are willing to pay for as opposed to that which they think should be freely 
available? Are sites designed with the user-based process different from those designed 
without user input? Are they more effective? 

The user-based design process should be readily transferable to other groups and 
other Web sites, and it has been described in sufficient detail in this paper to allow for use 
by other Web site designers. Only through additional use by other Web site designers in 
other situations can the transferability really be tested. 
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Notes 



1 . For positive features, the scale is 1 to 5 with 1 as “no influence at all” and 5 as “highly 
influential;” for negative features, the scale is 1 to 5 with 1 as “no negative influence at all” and 5 
as “most negative influence.” In both cases, a “5" represents the high extreme of a participant’s 
judgment. 

2. Some of this information is already on the Web, e.g. Web sites for corporations, which include 
information often contained in printed annual reports. 
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Appendix A. University of Maryland Human Subjects Research Approval 

UNIVERSITY OF MARYLAND 
College Park, MD 20742 



INSTITUTIONAL REVIEW BOARD 

APPLICATION FOR REVIEW OF RESEARCH USING HUMAN SUBJECTS 
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Name of Principal Investigator F'leen G. Abels 
(or Faculty Advisor) 
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Disclosure Inc. , . • 

rtas .^n^s project Peen approved by a department Human Subject Review Committee? 

Yes // JQ / 7 ( date ) No 

Please attach a copy of your responses to questions I - IV of the instructions 
along with the proposal. 

Please specify whether this research should be exempt or non-exempt for human 
subjects review and stipulate which of the five exemptions reasons (attached) 
justifies an exempt "status: 



Exempt y 



Non-Exempt 
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I. The College of Library and Information Services-proposes to perform research on user needs 
and to design a working prototype Web site for Disclosure’s academic business co mm u n ity. 

A focus group session will be held with business school faculty from the Washington and 
Baltimore areas to determine criteria for the development of Web pages. 

II. Twenty faculty members have been invited from business schools at UMCP, Georgetown 
University, George Washington University and University of Baltimore. We tried to get a mix of 
assistant, associate, and senior faculty members primarily in the areas of finance and accounting 
and including both male and female faculty members. Some of the selected faculty members 
have home pages and all must have electronic mail accounts. Invitations have been sent via e- 
mail. 

III. The subjects will participate in an electronic brainstorming session held in the AT&T 
teaching theater. They will be asked questions relating to what they look for in web pages and 
their use in general of information sources. 

IV. There are no risks to the subjects. 

V. The BrainQuest software being used in the session permits participation anonymously so the 
criteria developed will not be linked to any one individual. Furthermore, any reporting of the 
results of this study will not identify the participants. 

VI. Participants have been told the exact intent of the session. A copy of the invitation is 
attached. 
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Appendix B. Invitation to Participate 



As part of a research effort to identify criteria for the development of World Wide 
Web sites to serve the academic business community, we are convening a half-day focus 
session of selected business school faculty from the Washington and Baltimore areas. 

We would like to invite you to participate. The session will be held on Friday, January 
31, from 9:00 a.m. to 12:00 p.m. with lunch afterwards, and you will receive an 
honorarium of $100 for your participation. The session will be held in the AT& T 
Electronic Teaching Theater at the University of Maryland College Park, using the 
BrainQuest software. This research is funded by 
Disclosure, Inc. 

The following is the schedule for the day: 

9:00 a.m. - 9:30 a.m. Registration and Coffee 

9:30 a.m. - 10:45 a.m. Focus Group Brainstorming 

10:45 a.m. - 11:00 a.m. Break 

1 1:00 a.m. - 12:00 p.m. Setting Priorities 

12:15 p.m. - 1:30 p.m. Lunch at the Rossborough Inn 

The focus group session should be mutually beneficial. We are certain that you 
will provide us with valuable input for our project. In return, you will have a chance to 
network with faculty from other business schools in the area, plus receive the 
honorarium. 

Please let us know if you will be able to participate in this session by sending 
e-mail to ea29@umail.umd.edu no later than Thursday January 9, 1997. If you are 
unable to attend but know of another business school faculty member who may be 
interested, we would appreciate your suggestions. If you will be attending the session, 
we will mail a map and parking permit to you. We will be sending a very brief 
questionnaire via e-mail prior to the session to obtain needed demographic data. 

We look forward to hearing from you. 

Sincerely, 



Eileen G. Abels Marilyn Domas White 

Assistant Professor Associate Professor 
College of Library and Information Services 
University of Maryland, College Park 
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Appendix C. E-mail Questionnaire 



Dear [NAME]: 

Thank you for agreeing to participate in the Business Faculty Information Needs focus group 
session. I would appreciate it if you would complete the brief questionnaire that follows and 
return it via electronic mail. We will be using the information to structure our discussion 
questions and to avoid using time During the group session for gathering basic background 
information. Please make sure that you continue to the end of the questionnaire in your 
response. 

Indicate the frequency that most accurately describes your electronic mail usage under normal 
circumstances: (Place an X next to the most appropriate answer.) 

Daily 

Several times a week 

Several times a month 

Almost never 

Please estimate the month and year in which you first began to use electronic mail: 

Month Y ear 

Do you use the World Wide Web: 

No Yes 

If yes, indicate the frequency that most accurately describes your Web usage under normal 
circumstances: 

Daily 

Several times a week 

Several times a month 

Almost never 

Please estimate the month and year in which you first began to use the World Wide Web: 

Month 

Year 

Please indicate any of the following categories which describe functions for which you have used 
the World Wide Web: 

Research 

Teaching 

Consulting 

Administration 

Editorial work 

Other - Please describe: 
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Please indicate all the levels of students that you instruct: 

“ Undergraduate - ~~ - ~ - ■ — 

Masters 

Ph.D. 

Do you encourage your students to use the World Wide Web: 

Yes 

No 

Are there electronic forms of business information that you purchase or pay a fee for currently*? 
No 

Yes Please list: 

Are you familiar with the Disclosure company: 

No 

Yes Please explain: 
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Appendix D. Description of the VisionQuest Software 



The VisionQuest software product was originally produced by Intellect 
Corporation. Unfortunately, it is no longer supported or available for purchase. The 
Windows version of the software was used for the focus group session. Descriptions of 
this software are also available in published reviews (Keizer, 1995; Kranz, 1994) and a 
Web site (http://www.inform.umd. edu:8080/TT/Specs/Software/VQ.html). 

VisionQuest is designed to facilitate group interaction by providing functionalities 
for such group processes as idea generation, grouping, and evaluation. Brainwriting 
supports a brainstorming style of idea generating. Participants simultaneously contribute 
items anonymously and send them to the facilitator's station where they are collected and 
made available to the entire group. Group members can thus monitor others' suggestions 
while compiling their own contributions. Comments can be added to individual items by 
the use of comment cards which again offer individual anonymity combined with group 
display. Grouping of categories is facilitated by the use of compactor functionality. The 
compactor lets participants move items into categories. These may be pre-specified. 
VisionQuest provides a range of options for evaluating items once they have been 
generated. Group members can define the relative significance of items(the categorize 
function) , assign scores which can subsequently be normalized, vote, rate items with the 
scale being determined by the facilitator, or rank items by dragging them up or down a 
lists. Evaluation is performed individually and then the results are anonymously 
submitted to the facilitator's station where the software compiles the results and returns 
the results to the group. 

During the focus group session, the brainwriting capability was used to generate 
responses to questions regarding participants' information use and generate criteria 
indicative of good or poor design. The results from these open-ended questions were then 
evaluated using the rating functions and in some cases categorized by anticipated 
usefulness. Rating was done using five point scales oriented so that low ratings 
corresponded to least impact and high ratings corresponded to highest impact. 





Appendix E. E-Mail Evaluation of the Focus Group Session 



Dear [NAME] : 

Thank you again for participating in the focus group. We would appreciate it if you 
would take a few moments to fill out this evaluation form. Please indicate your assessment of the 
following areas using the 1-5 scale where one indicates complete disagreement and 5 indicates 
complete agreement. If you prefer, you could print out the evaluation and mail it back to me. 

Evaluation of Business Information Needs Focus Group Session 

A. I understand the purpose of the focus group session. 

1 Disagree completely 

2 

3 

4 

5 Agree completely 

B. The session was well organized. 

1 Disagree completely 

2 

3 

4 

5 Agree completely 

C. The questions discussed fit the goals of the session. 

1 Disagree completely 

2 

3 

4 

5 Agree completely 

D. Using the Vision Quest software made it easy for me to express my ideas and opinions. 

1 Disagree completely 

2 

3 

4 

5 Agree completely 

E. The session allowed me to express my real thoughts and opinions. 

1 Disagree completely 

2 

3 

4 

5 Agree completely 



48 



F . The structure of the session facilitated the emergence of the widest possible range of thoughts 

and Opinions? ~ ~ - - — - - - •- — - - - - — 

1 Disagree completely 

2 

3 

4 

5 Agree completely 

G. I benefited from participating in the session. 

1 Disagree completely 

2 

3 

4 

5 Agree completely 

H. I would be willing to participate in a future focus group of this type. 

1 Disagree completely 

2 

3 

4 

5 Agree completely 

I. Are there any other thoughts you would like to share about the focus group session? 
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Appendix F. Chronology of the Project 



Date 



Project Team Activities 



Events 



December 

1996 



January 

1997 



February 

1997 



March 1997 
April 1997 



May 1997 
June 1997 



October 

1997 



November 

1997 



Selection of participating business schools. 
Decision to schedule focus group in 
January, 1997. 

Selection of initial participants. 

Invitation developed. 

Questionnaire developed 

Focus group session questions developed. 

Facilities arrangements made. 

Participant biographies developed. 
Questionnaire responses compiled and 
analyzed. 

Focus group data analyzed. 

Evaluation questionnaire developed and 
sent. 

Questionnaire responses compiled and 
analyzed. 

Preliminary design of Web site. 

Wrote article reporting on needs 
assessment. 

Initial development of Web site continued. 

Web site development continued. 
Formative evaluation. 

Wrote article reporting on user-based Web 
site design process. 

Second stage of development of Web site. 

Did various iterations of design and 
expanded content. 

Began final development stage and 
prepared for transition of site to Disclosure. 



Finalized design. Checked for active sites. 



Project funding approved by 
Disclosure, Inc. 



Invitation issued to selected 
business school faculty. 

1/9 Initial deadline for response. 
1/22 Extended deadline for 
response. 

1/24 Questionnaire e-mailed to 
participants. 

1/3 1 Focus group met. 

2/19 Evaluation e-mailed to 
paricipants. 



4/8 Met with Disclosure staff to 
present initial design. 

4/15 Focus group members 
asked to evaluate initial design. 



6/26 Met with Disclosure staff 
for feedback from sales staff. 

10/9 Met with Disclosure staff 
to present development and 
agree upon final format and 
criteria. 

Finalized process for transfer. 
First article published in 
Internet Research. 
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-Date 



Project Team Activities 



Events 



December Sent zipped files to Disclosure. Disclosure begins process of 

1997 Wrote technical report. incorporating site for academics 

into their existing site. 
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